RESUMEN
Objective To investigate the performance of a commercially available artificial intelligence (AI) algorithm for detection of pulmonary embolism (PE) on contrast-enhanced CTs in patients hospitalized for COVID-19. Patients & Methods Retrospective analysis was performed of all contrast-enhanced chest CTs on patients admitted for COVID-19 between March 2020 and December 2021. Based on the original radiology reports, all PE-positive exams were included (n=527). Using a reversed flow single gate diagnostic accuracy case-control model, a randomly selected cohort of PE-negative exams (n=977) was included. Pulmonary parenchymal disease severity was assessed for all included studies using a semi-quantitative system, the Total Severity Score (TSS). All included CTs were sent for interpretation by the commercially available AI algorithm, Aidoc. Discrepancies between AI and original radiology reports were resolved by three blinded radiologists, who rendered a final determination of indeterminate, positive, or negative. Results A total of 78 studies were found to be discrepant, of which 13 (16.6%) were deemed indeterminate by readers and excluded. The sensitivity and specificity of AI was 93.2%;(95% confidence interval [CI] 90.6-95.2%), and 99.6%;(95% CI 98.9-99.9%), respectively. AI's accuracy for all TSS groups (mild, moderate, severe) was high (98.4%, 96.7%, and 97.2%, respectively). AI was more accurate in PE detection on CTPAs vs CECTs (P < .001), with optimal HU of 362 (P=.048). Conclusion The AI algorithm demonstrated high sensitivity, specificity, and accuracy for PE on contrast enhanced CTs in COVID-19 patients regardless of parenchymal disease. Accuracy was significantly affected by the mean attenuation of the pulmonary vasculature. How this affects the legitimacy of the binary outcomes reported by AI is not yet known.
RESUMEN
Objective: To investigate the performance of a commercially available artificial intelligence (AI) algorithm for the detection of pulmonary embolism (PE) on contrast-enhanced computed tomography (CT) scans in patients hospitalized for coronavirus disease 2019 (COVID-19). Patients and Methods: Retrospective analysis was performed of all contrast-enhanced chest CT scans of patients admitted for COVID-19 between March 1, 2020 and December 31, 2021. Based on the original radiology reports, all PE-positive examinations were included (n=527). Using a reversed-flow single-gate diagnostic accuracy case-control model, a randomly selected cohort of PE-negative examinations (n=977) was included. Pulmonary parenchymal disease severity was assessed for all the included studies using a semiquantitative system, the total severity score. All included CT scans were sent for interpretation by the commercially available AI algorithm, Aidoc. Discrepancies between AI and original radiology reports were resolved by 3 blinded radiologists, who rendered a final determination of indeterminate, positive, or negative. Results: A total of 78 studies were found to be discrepant, of which 13 (16.6%) were deemed indeterminate by readers and were excluded. The sensitivity and specificity of AI were 93.2% (95% CI, 90.6%-95.2%) and 99.6% (95% CI, 98.9%-99.9%), respectively. The accuracy of AI for all total severity score groups (mild, moderate, and severe) was high (98.4%, 96.7%, and 97.2%, respectively). Artificial intelligence was more accurate in PE detection on CT pulmonary angiography scans than on contrast-enhanced CT scans (P<.001), with an optimal Hounsfield unit of 362 (P=.048). Conclusion: The AI algorithm demonstrated high sensitivity, specificity, and accuracy for PE on contrast-enhanced CT scans in patients with COVID-19 regardless of parenchymal disease. Accuracy was significantly affected by the mean attenuation of the pulmonary vasculature. How this affects the legitimacy of the binary outcomes reported by AI is not yet known.
RESUMEN
We introduce a multi-institutional data harvesting (MIDH) method for longitudinal observation of medical imaging utilization and reporting. By tracking both large-scale utilization and clinical imaging results data, the MIDH approach is targeted at measuring surrogates for important disease-related observational quantities over time. To quantitatively investigate its clinical applicability, we performed a retrospective multi-institutional study encompassing 13 healthcare systems throughout the United States before and after the 2020 COVID-19 pandemic. Using repurposed software infrastructure of a commercial AI-based image analysis service, we harvested data on medical imaging service requests and radiology reports for 40,037 computed tomography pulmonary angiograms (CTPA) to evaluate for pulmonary embolism (PE). Specifically, we compared two 70-day observational periods, namely (i) a pre-pandemic control period from 11/25/2019 through 2/2/2020, and (ii) a period during the early COVID-19 pandemic from 3/8/2020 through 5/16/2020. Natural language processing (NLP) on final radiology reports served as the ground truth for identifying positive PE cases, where we found an NLP accuracy of 98% for classifying radiology reports as positive or negative for PE based on a manual review of 2,400 radiology reports. Fewer CTPA exams were performed during the early COVID-19 pandemic than during the pre-pandemic period (9806 vs. 12,106). However, the PE positivity rate was significantly higher (11.6 vs. 9.9%, p < 10-4) with an excess of 92 PE cases during the early COVID-19 outbreak, i.e., ~1.3 daily PE cases more than statistically expected. Our results suggest that MIDH can contribute value as an exploratory tool, aiming at a better understanding of pandemic-related effects on healthcare.